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(54) Signal processing apparatus, signal processing method of the apparatus, operation process 
program of the method, and storage medium storing the program 



(57) It is an object of this invention to appropriately 
interpolate an image signal having a frequency close to 
the Nyquist frequency of an image sensing device. To 
achieve this object, a signal processing apparatus in- 
eludes a first interpolation circuit for performing interpo- 
lation in a pixel position as an object of interpolation on 
the basis of pixel data in first and second directions, a 
second Interpolation circuit for performing interpolation 
in a pixel position as an object of interpolation on the 
basis of pixel data in the first direction, a third interpola- 
tion circuit for performing interpolation in a pixel position 



as an object of interpolation on the basis of pixel data in 
the second direction, a determination circuit for deter- 
mining correlations in the first and second directions 
with respect to a signal at a pixel position as an object 
of interpolation, a selection circuit for selecting interpo- 
lated data based on an output from one of the second 
and third interpolation circuits, on the basis of the deter- 
mination result from the determination circuit, and an 
output unit for outputting an interpolated signal on the 
basis of interpolated data, which is based on an output 
from the first interpolation circuit, and the output inter- 
polated data from the selection circuit. 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to a signal 
processing apparatus which is suitably used as, e.g., a 
digital still camera and which generates a luminance sig- 
nal or a color signal from an output from an image sens- 
ing device having a plurality of different types of filters, 
and to a signal processing method of the apparatus. 

BACKGROUND OF THE INVENTION 

[0002] In a conventional single-sensor camera using 
an image sensing device such as a CCD, a color filter 
of one of a plurality of types (e.g., one of three colors R. 
G, and B) is formed for each pixel of the image sensing 
device. For example, the Bayer arrangement shown in 
FIG. 1 is well known. 

[0003] In a single-sensor camera having this Bayer 
arrangement, color information of only one of R, G, and 
B is obtained by each pixel. To obtain color information 
of all of R, G, and B in each pixel, therefore, interpolation 
must be performed in a subsequent circuit. The simplest 
interpolation method is to set 0 as all color signal levels 
except for a color signal to be interpolated, and perform 
two-dimensional, low-pass filter (LPF) processing for 
each pixel, thereby obtaining an interpolated value. 
[0004] Unfortunately, if this two-dimensional LPF 
processing alone is simply performed as described 
above, high-frequency components are lost from the fre- 
quency characteristics of the interpolated image, and 
this blurs the image. 

[0005] To solve this problem, a method by which the 
LPF processing is changed in accordance with the char- 
acteristics of an image has been proposed. For exam- 
ple, if a certain portion of an image is found to be a ver- 
tical stripe, the LPF processing is performed only in the 
lateral direction of the image; if a certain portion of an 
image is found to be a lateral stripe, the LPF processing 
is performed only in the longitudinal direction of the im- 
age. If a certain portion of an image is found to be a 
wave intermediate between vertical and lateral stripes, 
the LPF processing in the lateral direction of the image 
and the LPF processing in the longitudinal direction of 
the image are weighted in accordance with the degree 
of the wave, thereby interpolating a color signal. 
[0006] However, the above conventional method 
does not take into account the sensing of an image (e. 
g. , a vertically striped pattern or laterally striped pattern) 
having a frequency close to the Nyquist frequency of the 
image sensing device. 

[0007] A frequency close to the Nyquist frequency of 
the image sensing device is a folded signal, so it is dif- 
ficult to determine whether the image is a vertical or lat- 
eral stripe. This may lead to erroneous determination 
between vertical and lateral stripes. For example, al- 
though an actual object is a lateral stripe, the longitudi- 



nal LPF processing may be performed by mistake for a 
certain pixel to be interpolated, whereas the lateral LPF 
processing is performed for the next pixel to be interpo- 
lated. That is, appropriate color signal interpolation may 
5 not be performed continuously. 

SUMMARY OF THE INVENTION 

[0008] The present invention has been made in con- 
sideration of the above prior art, and has as its object to 
provide a signal processing apparatus or signal 
processing method capable of performing appropriate 
color signal interpolation even when an image signal 
close to the Nyquist frequency of an image sensing de- 
vice exists. 

[0009] To achieve the above object, according to an 
invention of the present application, a signal processing 
apparatus for interpolating an output color signal from 
an image sensing device is characterized by comprising 
first interpolating device that performs interpolation in a 
pixel position as an object of interpolation on the basis 
of pixel data in first and second directions, second inter- 
polating device that performs interpolation in a pixel po- 
sition as an object of interpolation on the basis of pixel 
data in the first direction, third interpolating device that 
performs interpolation in a pixel position as an object of 
interpolation on the basis of pixel data in the second di- 
rection, determining device that determines correlations 
in the first and second directions with respect to a signal 
at a pixel position as an object of interpolation, selecting 
device that selects interpolated data based on an output 
from one of the second and third interpolating device, 
on the basis of the determination result from the deter- 
mining device, and output device that outputs an inter- 
polated signal on the basis of interpolated data, which 
is based on an output from the first interpolating device, 
and the output interpolated data from the selecting de- 
vice. 

[0010] According to another invention of the present 
application, a signal processing method of interpolating 
an output color signal from an image sensing device is 
characterized by comprising the first interpolation step 
of performing interpolation in a pixel position as an ob- 
ject of interpolation on the basis of pixel data in first and 
second directions, the second interpolation step of per- 
forming interpolation in a pixel position as an object of 
interpolation on the basis of pixel data in the first direc- 
tion, the third interpolation step of performing interpola- 
tion in a pixel position as an object of interpolation on 
the basis of pixel data in the second direction, the se- 
lection step of determining correlations in the first and 
second directions with respect to a signal at a pixel po- 
sition as an object of interpolation and, on the basis of 
the determination result, selecting interpolated data 
based on an output from one of the second and third 
interpolation steps, and the output step of outputting an 
interpolated signal on the basis of interpolated data, 
which is based on an output from the first interpolation 
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step, and the output interpolated data from the selection 
step. 

[0011] According to still another invention of the 
present application, in a storage medium storing an op- 
eration process program for interpolating an output color 
signal from an image sensing device, the operation 
process program comprises the first interpolation step 
of performing interpolation in a pixel position as an ob- 
ject of interpolation on the basis of pixel data in first and 
second directions, the second interpolation step of per- 
forming interpolation in a pixel position as an object of 
interpolation on the basis of pixel data in the first direc- 
tion, the third interpolation step of performing interpola- 
tion in a pixel position as an object of interpolation on 
the basis of pixel data in the second direction, the se- 
lection step of determining correlations in the first and 
second directions with respect to a signal at a pixel po- 
sition as an object of interpolation and, on the basis of 
the determination result, selecting interpolated data 
based on an output from one of the second and third 
interpolation steps, and the output step of outputting an 
interpolated signal on the basis of interpolated data, 
which is based on an output from the first interpolation 
step, and the output interpolated data from the selection 
step. 

[0012] According to still another invention of the 
present application, an operation process program for 
interpolating an output color signal from an image sens- 
ing device comprises the first interpolation step of per- 
forming interpolation in a pixel position as an object of 
interpolation on the basis of pixel data in first and second 
directions, the second interpolation step of performing 
interpolation in a pixel position as an object of interpo- 
lation on the basis of pixel data in the first direction, the 
third interpolation step of performing interpolation in a 
pixel position as an object of interpolation on the basis 
of pixel data in the second direction, the selection step 
of determining correlations in the first and second direc- 
tions with respect to a signal at a pixel position as an 
object of interpolation and, on the basis of the determi- 
nation result, selecting interpolated data based on an 
output from one of the second and third interpolation 
steps, and the output step of outputting an interpolated 
signal on the basis of interpolated data, which is based 
on an output from the first interpolation step, and the 
output interpolated data from the selection step. 
[0013] Other objects and advantages besides those 
discussed above shall be apparent to those skilled in 
the art from the description of a preferred embodiment 
of the invention which follows. In the description, refer- 
ence is made to accompanying drawings, which form a 
part thereof, and which illustrate an example of the in- 
vention. Such example, however, is not exhaustive of 
the various embodiments of the invention, and therefore 
reference is made to the claims which follow the descrip- 
tion for determining the scope of the invention. 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0014] 

s Fig. 1 is a view showing the Bayer arrangement as 
a color filter arrangement in a single-sensor image 
sensing device; 

Fig. 2 is a block diagram showing an outline of a 
signal processing apparatus according to the first 
10 embodiment; 

Fig. 3 is a view showing a color filter arrangement 
in an image sensing device according to the first 
embodiment; 

Fig. 4 is a view showing an offset sampling arrange- 
rs ment as a color filter arrangement in a single-sensor 
image sensing device; 

Fig. 5 is an operation process flow chart of the signal 
processing apparatus according to the first embod- 
iment of the present invention; 
Fig. 6 is a block diagram showing an outline of a 
signal processing apparatus according to the sec- 
ond embodiment; 

Figs. 7A and 7B are views showing examples of 
HPF in the second embodiment; 
Fig. 8 is a block diagram showing an outline of a 
signal processing apparatus according to the third 
embodiment; 

Fig. 9 is a block diagram showing an outline of a 
signal processing apparatus according to the fourth 
embodiment; and 

Fig. 1 0 is a graph for explaining a positive/negative 
determination gain circuit 19 of the fourth embodi- 
ment. 



[0015] Preferred embodiments of the present inven- 
tion will now be described in detail in accordance with 
the accompanying drawings. Note that each embodi- 
ment will be explained by using the Bayer arrangement 
shown in Fig. 1 as a color filter arrangement of an image 
sensing device. However, another color filter arrange- 
ment is also usable. 

(First Embodiment) 

[0016] Fig. 2 is a block diagram showing an outline of 
the arrangement of a signal processing apparatus ac- 
cording to the first embodiment. 
[0017] Referring to Fig. 2, a WB (White Balance) cir- 
cuit 1 corrects the white balance of an image signal. This 
WB circuit 1 receives an image signal which is output 
from an image sensing device and A/D-converted. 
[001 8] AG interpolation circuit 2 performs G signal in- 
terpolation for each pixel. Likewise, an R interpolation 
circuit 15 performs R signal interpolation, and a B inter- 
polation circuit 16 performs B signal interpolation. 
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[001 9] A vertical/lateral determination circuit 1 0 deter- 
mines whether an image is a vertical stripe, a lateral 
stripe, or an intermediate between them. An absolute 
value formation circuit 13 changes an output parameter 
signal from the vertical/lateral determination circuit 10 
into an absolute value. Note that the processes in the 
vertical/lateral determination circuit 1 0 and the absolute 
value formation circuit 13 will be described later. 
[0020] An APC circuit 14 corrects the aperture of the 
output G signal from the G interpolation circuit 2. This 
APC circuit for performing aperture correction is also 
placed in the subsequent stage of each of the R inter- 
polation circuit 15 and the B interpolation circuit 16. 
[0021] By the above interpolation processes, all of R, 
G, and B signals corresponding to individual pixel posi- 
tions are obtained. In addition, a luminance signal Y is 
obtained by the subsequent process in accordance with 

Y = 0.3R + 0.59G-i-0.11B 

[0022] For the sake of descriptive simplicity, only in- 
terpolation for a G signal will be explained in the first and 
other embodiments. Interpolation for R and B signals is 
linearly performed. 

[0023] The internal configuration of the G interpola- 
tion circuit 2 will be explained below. 
[0024] A 0 insertion circuit 3 sets 0 as the output signal 
levels of pixels other than a pixel having a color to be 
interpolated, i.e., a pixel having a G filter. 
[0025] A two-dimensional LPF 4 interpolates an im- 
age signal which is close to the Nyquist frequency and 
which is not easily found to be a vertical stripe or lateral 
stripe. This two-dimensional LPF 4 performs interpola- 
tion by using pixels in a two-dimensional range very 
close to the position of a pixel to be interpolated, so that 
no image blur occurs. 

[0026] A horizontal LPF (Low Pass Filter) 5 performs 
interpolation by using an image signal in the lateral di- 
rection (horizontal direction) with respect to the position 
of a pixel to be interpolated. In this embodiment, this 
horizontal LPF 5 performs interpolation by using an im- 
age signal within a range wider than that of the image 
signal used by the two-dimensional LPF 4, in the hori- 
zontal direction of the position of a pixel to be interpo- 
lated. 

[0027] A vertical LPF 6 performs interpolation by us- 
ing an image signal in the longitudinal direction with re- 
spect to the position of a pixel to be interpolated. In this 
embodiment, this vertical LPF 6 performs interpolation 
by using an image signal within a range wider than that 
of the image signal used by the two-dimensional LPF 4 : 
in the vertical direction of the position of a pixel to be 
interpolated. 

[0028] An adder 7 calculates the difference between 
the output from the two-dimensional LPF 4 and the out- 
put from the horizontal LPF 5. An adder 8 calculates the 
difference between the output from the two-dimensional 



LPF 4 and the output from the vertical LPF 6. A switch 
9 selects the output from the adder 7 or 8 on the basis 
of the determination result from the vertical/lateral de- 
termination circuit 10. A multiplier 11 multiplies the sig- 
5 nal output selected by the switch 9 by the output from 
the absolute value formation circuit 13. An adder 12 
adds the output from the two-dimensional LPF 4 and the 
output from the multiplier 11 . 

[0029] Fig. 5 is an operation process flow chart con- 
10 ceming image signal (G signal) interpolation in the first 
embodiment. Fig. 3 is a view clearly showing pixel po- 
sitions in the color filter arrangement shown in Fig. 1 . 
The interpolation process of this embodiment will be de- 
scribed below with reference to Figs. 3 and 5. 
is [0030] First, in S1 01 , the 0 insertion circuit 3 inserts 0 
in pixels corresponding to color filters except forG, i.e., 
color filters of R and B. 

[0031] The flow advances to S102, and the two-di- 
mensional LPF 4 performs pixel interpolation forthe out- 
20 put signal from the 0 insertion circuit 3, thereby obtaining 
a two-dimensional G interpolated signal. This two-di- 
mensional LPF 4 is so configured as to perform interpo- 
lation of [1 21 ]/2 by using pixel data in the vertical direc- 
tion and interpolation of [1 2 1]/2 by using pixel data in 
25 the horizontal direction. For example, an interpolated 
value G33' at a pixel position R33 shown in Fig. 3 is in 
the vertical direction 

3Q (1 x G32 + 2 x R33 + 1 x G34)/2 = (G32 + G34)/2 

and in the horizontal direction 

35 (1 X G23 + 2 x R33 + 1 X G43)/2 = (G23 + G43)/2 

Therefore, by calculating the average, 

4Q G33' = (G32 + G34 + G23 + G43)/4 

[0032] The flow advances to S103, and the vertical/ 
lateral determination circuit 10 checks whether the pixel 
signal to be interpolated has a large correlation in the 
45 vertical direction or in the horizontal direction. More spe- 
cifically, the vertical/lateral determination circuit 10 
checks whether the image is a vertically striped pattern 
or a laterally striped pattern. 

[0033] The determination method in this vertical/lat- 
50 eral determination circuit will be described below. As- 
sume, for example, that the position of a pixel to be in- 
terpolated is R33. 



diffHV33 = diffH33 - diffV33 
diffH33 = IG32 - G34I + 12R33 - R31 - R35I 
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diffV33 = IG23 - G43I + I2R33 - R13 - R53I 

[0034] Note that diffH indicates a level difference be- 
tween pixels, in the horizontal direction, centering 
around the pixel to be interpolated, and diffV indicates 
a level difference between pixels, in the vertical direc- 
tion, centering around the pixel to be interpolated. Note 
also that diffHV indicates the difference of the level dif- 
ference between the pixels in the vertical direction from 
the level difference between the pixels in the horizontal 
direction. That is, if diffHV is positive, the image is a ver- 
tically striped pattern; if diffHV is negative, the image is 
a laterally striped pattern. The value of diffHV is set with- 
in the range of -1 to 1 by clipping the calculated value 
by a certain threshold value and normalizing this clipped 
value. Note that an expression for calculating diffHV is 
not restricted to the above one. 
[0035] If diffHV > 0, i.e., if the image is found to be a 
vertically striped pattern, the flow advances to S104. If 
diffHV < 0, i.e., if the image is found to be a laterally 
striped pattern, the flow advances to S105. 
[0036] In S1 04, the switch 9 selects an output signal 
Dv from the adder 8. Note that the vertical LPF 6 is so 
configured as to perform interpolation of [1 4 6 4 1]/8 by 
using pixel data in the vertical direction. For example, 
when the pixel position R33 is the object of interpolation, 
an output value Gv33 from the vertical LPF 6 is 

Gv33 = (1 x R13 + 4 x G23 + 6 x R33 + 4 x G43 + 1 
x R53)/8 
= (G23 + G43)/2 

[0037] Accordingly, when the pixel position R33 is the 
object of interpolation, the output difference signal Dv 
from the adder 8 is 

Dv33 = Gv33 - G33' = (G23 + G43 - G32 - G34)/4 

[0038] I n S 1 05, on the other hand, the switch 9 selects 
an output signal Oh from the adder 7. Note that the hor- 
izontal LPF 5 is so configured as to perform interpolation 
of [1 4 6 4 1]/8 by using pixel data in the horizontal di- 
rection. For example, when the pixel position R33 is the 
object of interpolation, an output value Gh33 from the 
horizontal LPF 5 is 

Gh33 = (1 x R31 + 4 x G32 + 6 x R33 + 4 x G34 + 1 
x R35)/8 
= (G32 + G34)/2 



[0039] Accordingly, when the pixel position R33 is the 
object of interpolation, the output difference signal Dh 
from the adder 7 is 

Dh33 = Gh33 - G33' = (G32 + G34 - G23 - G43)/4 

[0040] In S106, the multiplier 11 multiplies the signal 
selected by the switch 9 by the output from the absolute 
10 value formation circuit 13. This absolute value formation 
circuit 13 outputs a signal as the absolute value of diff- 
HV. In S107, the output from the two-dimensional LPF 
4 and the output from the multiplier 1 1 are added to out- 
put a final interpolated signal G. 
15 [0041] By the processing explained above, appropri- 
ate interpolation is performed. For example, if it is diffi- 
cult to distinguish between vertical and lateral stripes 
owing to the influence of an image signal near the 
Nyquist frequency, the value of diffHV is small, so the 
20 multiplier 11 decreases the value of Gh or Gv. That is, 
the output interpolated data from the two-dimensional 
LPF 4 is weighted by the output interpolated data Gh or 
Gv. This suppresses the influence of a folded signal and 
makes ft possible to perform appropriate interpolation. 
25 [0042] Also, if diffHV = 1 , i.e., if an object is found to 
be a vertically striped pattern, the switch 9 selects Dv, 
and this Dv is multiplied by 111. As a consequence, the 
output interpolated data from the multiplier 11 is weight- 
ed. When the pixel position R33 is the object of interpo- 
30 lation, for example, the output (i.e.. the final interpolated 
signal G) from the adder 1 2 is 

G33 = G33' + Dv33 = (G23 + G43)/2 

35 

This prevents deterioration of the characteristics in the 
longitudinal direction. 

[0043] If diffHV = -1 , i.e., if the object is found to be a 
laterally striped pattern, the switch 9 selects Dh , and this 
40 Dh is multiplied by 1-11. Consequently, the output inter- 
polated data from the multiplier 11 is weighted. When 
the pixel position R33 is the object of interpolation, for 
example, the output (i.e., the final interpolated signal G) 
from the adder 1 2 is 

45 

G33 = G33' + Dh33 = (G32 + G34)/2 

This prevents deterioration of the characteristics in the 
so longitudinal direction (vertical direction) of the image. 

(Second Embodiment) 

[0044] Fig. 6 is a block diagram of a signal processing 
55 apparatus in the second embodiment. This configura- 
tion shown in Fig. 6 is equivalent to a configuration in 
which the horizontal LPF 5 and the adder 7 shown in 
Fig. 2 are replaced with a vertical H PF (High Pass Filter) 



25 



55 



5 



9 



EP 1 229 743 A2 



10 



25 and the vertical LPF 6 and the adder 8 shown in Fig. 
2 are replaced with a horizontal HPF 26. The same ref- 
erence numerals as in Fig. 2 denote parts which perform 
the same processes, and a detailed description thereof 
will be omitted. 

[0045] The vertical HPF 25 has a filter characteristic 
shown in Fig. 7A. That is, the output from this vertical 
HPF 25 is the same as the signal of the difference be- 
tween the output from the horizontal LPF 5 and the out- 
put from the two-dimensional LPF in the first embodi- 
ment. For example, when a pixel position R33 shown in 
Fig. 3 is the object of interpolation, an output signal Dh 
from the vertical HPF 25 is 

Dh33 = Gh33 - G33' = (G32 + G34 - G23 - G43)/4 

[0046] The horizontal HPF 26 has a filter characteris- 
tic shown in Fig. 7B. That is, the output from this hori- 
zontal HPF 26 is the same as the signal of the difference 
between the output from the vertical LPF 6 and the out- 
put from the two-dimensional LPF in the first embodi- 
ment. For example, when the pixel position R33 shown 
in Fig. 3 is the object of interpolation, an output signal 
Dv from the horizontal HPF 26 is 

Dv33 = Gv33 - G33' = (G23 + G43 - G32 - G34)/4 

[0047] This second embodiment can also achieve the 
same effects as in the first embodiment. 

(Third Embodiment) 

[0048] Fig. 8 is a block diagram showing the arrange- 
ment of a signal processing apparatus according to the 
third embodiment. This signal processing apparatus of 
the third embodiment has a configuration in which a gain 
circuit 17 and an adder 18 are added to the signal 
processing apparatus of the first embodiment shown in 
Fig. 2. In this block diagram shown in Fig. 8, the same 
reference numerals as in Fig. 2 denote parts which per- 
form the same processes, so a detailed description 
thereof will be omitted. 

[0049] The gain circuit 1 7 amplifies an output signal 
from a multiplier 1 1 . The output signal from this gain cir- 
cuit 1 7 is added to an output image signal from an APC 
circuit 14 by the adder 1 8. That is, the output difference 
signal from the multiplier 11 is a high-frequency signal 
adaptively selected in accordance with the pattern of an 
image. For example, when an image is a vertically or 
laterally striped pattern, the output level from the multi- 
plier is large, so the high-frequency signal is empha- 
sized; if it is impossible to identify the pattern of an im- 
age, emphasis of the high-frequency signal is sup- 
pressed. That is, adaptive favorable emphasis can be 
performed for the high-frequency portion of an interpo- 
lated signal. 



[0050] Also, as described above, the output from the 
multiplier 11 is used as a signal for emphasizing the 
high-frequency component of an image. That is, in this 
embodiment a G interpolation circuit 2 is also used as 
5 a circuit for emphasizing the high-frequency signal of an 
image. This simplifies the circuit configuration. 

(Fourth Embodiment) 

io [0051 ] Fig. 9 is a block diagram showing the arrange- 
ment of a signal processing apparatus according to the 
fourth embodiment. This signal processing apparatus of 
the fourth embodiment has a configuration in which a 
gain circuit 17, an adder 18, a positive/negative deter- 
mination gain circuit 1 9, an absolute value formation cir- 
cuit 20, and an adder 21 are added to the signal process- 
ing apparatus of the first embodiment shown in Fig. 2. 
In this block diagram shown in Fig. 9, the same refer- 
ence numerals as in Fig. 2 denote parts which perform 
the same processes, so a detailed description thereof 
will be omitted. 

[0052] In the fourth embodiment, a determination sig- 
nal diffHV from a vertical/lateral determination circuit 1 0 
is blanched. One determination signal is processed in 
the same manner as in the first embodiment. The other 
determination signal is supplied to the positive/negative 
determination gain circuit 1 9 where a desired gain is ap- 
plied only to a positive or negative signal. 
[0053] For example, when a twofold gain is applied 
only to a positive signal, the range of the determination 
signal which is from -1 to 1 changes to -1 to 2 as shown 
in Fig. 10. 

[0054] The absolute value f ormation circuit 20 chang- 
es this determination signal into an absolute value. The 
gain circu it 1 7 emphasizes the signal, and the adder 1 8 
adds the signal to a G interpolated signal. The result is 
a G interpolated signal in which the degree of high fre- 
quency emphasis in the horizontal direction differs from 
that in the vertical direction. 

(Other Embodiments) 

[0055] The present invention is also applicable to an 
image sensing device having a structure in which pixels 
are offset by a half pixel pitch every horizontal line as 
shown in Fig. 4. In this structure, the processing of each 
of the above embodiments can be implemented by re- 
placing the vertical correlation processing with oblique 
correlation processing. 

[0056] As an example, the present invention can be 
achieved by supplying program codes of software for 
implementing thef unctions of the above-mentioned em- 
bodiments to a signal processing apparatus via a net- 
work such as the Internet, and reading out and execut- 
ing the program codes stored in a storage medium by a 
computer (or a CPU or MPU) of the signal processing 
apparatus. 

[0057] In this case, the program codes read out from 
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the storage medium implement the functions of the 
above embodiments, and the storage medium storing 
the program codes constitutes the invention. 
[0058] As this storage medium for supplying the pro- 
gram codes, it is possible to use, e.g. , a floppy disk, hard 
disk, optical disk, magnetooptical disk, CD-ROM, CD-R, 
magnetic tape, nonvolatile memory card, and ROM. 
[0059] Also, besides the functions of the above em- 
bodiments are implemented by executing the readout 
program codes by the computer, the present invention 
includes a case where an OS (Operating System) or the 
like running on the computer performs part or the whole 
of actual processing in accordance with designations by 
the program codes and thereby implements the func- 
tions of the above embodiments. 
[0060] Furthermore, the present invention also in- 
cludes a case where the program codes read out from 
the storage medium are written in a memory of a func- 
tion extension board inserted into the computer or of a 
function extension unit connected to the computer, and, 
in accordance with designations by the program codes, 
a CPU or the like of the function extension board or func- 
tion extension unit performs part or the whole of actual 
processing and thereby implements the functions of the 
above embodiments. 

[0061] When the present invention is applied to the 
storage medium described above, this storage medium 
stores program codes corresponding to the flow charts 
explained previously. Briefly, modules essential to the 
signal processing apparatus of the present invention are 
stored in the storage medium. 

[0062] In the above embodiments as described 
above, on the basis of correlations of directions (e.g.. 
horizontal and vertical directions) with respect to the po- 
sition of a pixel to be interpolated, it is checked whether 
to give importance to data interpolated on the basis of 
pixel data in two-dimensional directions, or to data in- 
terpolated on the basis of pixel data in one direction. 
Therefore, when an image is found to be a vertically 
striped pattern or a laterally striped pattern, the frequen- 
cy characteristics of the image do not deteriorate. Also, 
even if an image can be either a vertically striped pattern 
or a laterally striped pattern, no adverse influence is ex- 
erted by a folded signal or the like. 
[0063] The present invention is not limited to the 
above embodiments and various changes and modifi- 
cations can be made within the spirit and scope of the 
present invention . Therefore, to apprise the public of the 
scope of the present invention the following claims are 
made. 

[0064] It is an object of this invention to appropriately 
interpolate an image signal having a frequency close to 
the Nyquist frequency of an image sensing device. To 
achieve this object, a signal processing apparatus in- 
cludes a first interpolation circuit for performing interpo- 
lation in a pixel position as an object of interpolation on 
the basis of pixel data in first and second directions, a 
second interpolation circuit for performing interpolation 



in a pixel position as an object of interpolation on the 
basis of pixel data in the first direction, a third interpola- 
tion circuit for performing interpolation in a pixel position 
as an object of interpolation on the basis of pixel data in 
5 the second direction, a determination circuit for deter- 
mining correlations in the first and second directions 
with respect to a signal at a pixel position as an object 
of interpolation, a selection circuit for selecting interpo- 
lated data based on an output from one of the second 
10 and third interpolation circuits, on the basis of the deter- 
mination result from the determination circuit, and an 
output unit for outputting an interpolated signal on the 
basis of interpolated data, which is based on an output 
from the first interpolation circuit, and the output inter- 
ns polated data from the selection circuit. 



Claims 

20 1. A signal processing apparatus for interpolating an 
output color signal from an image sensing device, 
comprising: 



first interpolating device that performs interpo- 
25 lation in a pixel position as an object of interpo- 

lation on the basis of pixel data in first and sec- 
ond directions; 

second interpolating device that performs inter- 
polation in a pixel position as an object of inter- 
30 polation on the basis of pixel data in the first 

direction; 

third interpolating device that performs interpo- 
lation in a pixel position as an object of interpo- 
lation on the basis of pixel data in the second 

35 direction; 

determining device that determines correla- 
tions in the first and second directions with re- 
spect to a signal at a pixel position as an object 
of interpolation; 

40 selecting device that selects interpolated data 

based on an output from one of said second 
and third interpolating device, on the basis of 
the determination result from said determining 
device; and 

45 output device that outputs an interpolated sig- 

nal on the basis of interpolated data, which is 
based on an output from said first interpolating 
device, and the output interpolated data from 
said selecting device. 

50 

2. The apparatus according to claim 1 , wherein on the 
basis of the determination result from said deter- 
mining device, said output device outputs an inter- 
polated signal by weighting the interpolated data 
55 based on the output from said first interpolating de- 
vice and the output interpolated data from said se- 
lecting device. 
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3. The apparatus according to claim 2, wherein if said 
determining device is expected to determine that 
the correlations in the first and second directions 
would become equal, said output device generates 
an interpolated signal by performing weighting such 
that the interpolated data based on the output from 
said first interpolating device is weighted more than 
the output interpolated data from said selecting de- 
vice. 

4. The apparatus according to claim 1, wherein said 
second interpolating device performs interpolation 
on the basis of pixel data, in the first direction, over 
a broader range than that when said first interpolat- 
ing device performs interpolation. 

5. The apparatus according to claim 1 , wherein said 
third interpolating device performs interpolation on 
the basts of pixel data, in the second direction, over 
a broader range than that when said first interpolat- 
ing device performs interpolation. 

6. The apparatus according to claim 1 , wherein said 
selecting device selects one of a difference be- 
tween the output interpolated data from said second 
interpolating device and the output interpolated da- 
ta from said first interpolating device, and a differ- 
ence between the output interpolated data from 
said third interpolating device and the output inter- 
polated data from said first interpolating device. 

7. The apparatus according to claim 6, wherein if said 
determining device determines that a pixel position 
as an object of interpolation has a higher correlation 
in the second direction than in the first direction, 
said selecting device selects a difference between 
the output interpolated data from said second inter- 
polating device and the output interpolated data 
from said first interpolating device. 

8. The apparatus according to claim 6, wherein if said 
determining device determines that a pixel position 
as an object of interpolation has a higher correlation 
in the first direction than in the second direction, 
said selecting device selects a difference between 
the output interpolated data from said third interpo- 
lating device and the output interpolated data from 
said first interpolating device. 

9. The apparatus according to claim 1, further com- 
prising emphasizing means for emphasizing the 
output interpolated signal from said output device 
in accordance with the determination result from 
said determining device. 

10. The apparatus according to claim 9, wherein said 
emphasizing device emphasizes one of the first and 
second directions. 



11. The apparatus according to claim 1 , wherein said 
second and third interpolating device are HPFs 
(High Pass Filters). 

s 12. The apparatus according to claim 1, wherein said 
image sensing device has color filters having a Bay- 
er arrangement. 

13. The apparatus according to claim 1 , wherein in said 
10 image sensing device, pixels are offset. 

14. A signal processing method of interpolating an out- 
put color signal from an image sensing device, com- 
prising: 

the first interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in first and 
second directions; 

the second interpolation step of performing in- 
terpolation in a pixel position as an object of in- 
terpolation on the basis of pixel data in the first 
direction; 

the third interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in the second 
direction; 

the selection step of determining correlations in 
the first and second directions with respect to 
a signal at a pixel position as an object of inter- 
polation and, on the basis of the determination 
result, selecting interpolated data based on an 
output from one of the second and third inter- 
polation steps: and 

the output step of outputting an interpolated 
signal on the basis of interpolated data, which 
is based on an output from the first interpolation 
step, and the output interpolated data from the 
selection step. 

15. The method according to claim 14, wherein on the 
basis of the determination result of the correlations 
in the first and second directions with respect to the 
signal at the pixel position as an object of interpo- 
lation, the output step comprises outputting an in- 
terpolated signal by weighting the interpolated data 
based on the output from the first interpolation step 
and the output interpolated data from the selection 
step. 

16. The method according to claim 15, wherein if it is 
expected to be determined that the correlations in 
the first and second directions would become equal, 
the output step comprises generating an interpolat- 
ed signal by performing weighting such that the in- 
terpolated data based on the output from the first 
interpolation step is weighted more than the output 
interpolated data from the selection step. 
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1 7. The method according to claim 1 4, wherein the sec- 
ond interpolation step comprises performing inter- 
polation on the basis of pixel data, in the first direc- 
tion, over a broader range than that when the first 
interpolation step performs interpolation. 

18. The method according to claim 1 4, wherein the third 
interpolation step comprises performing interpola- 
tion on the basis of pixel data, in the second direc- 
tion, over a broader range than that when the first 
interpolation step performs interpolation. 

19. The method according to claim 14, further compris- 
ing the emphasizing step of emphasizing the output 
interpolated signal from the output step in accord- 
ance with the determination result of the correla- 
tions in the first and second directions with respect 
to the signal at the pixel position as an object of in- 
terpolation. 

20. The method according to claim 1 9, wherein the em- 
phasizing step comprises emphasizing one of the 
first and second directions. 

21. The method according to claim 14, wherein the sec- 
ond and third interpolation steps comprise perform- 
ing interpolation by using HPFs (High Pass Filters). 

22. A storage medium storing an operation process 
program for interpolating an output color signal from 
an image sensing device, said operation process 
program comprising: 

the first interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in first and 
second directions; 

the second interpolation step of performing in- 
terpolation in a pixel position as an object of in- 
terpolation on the basis of pixel data in the first 
direction; 

the third interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in the second 
direction; 

the selection step of determining correlations in 
the first and second directions with respect to 
a signal at a pixel position as an object of inter- 
polation and, on the basis of the determination 
result, selecting interpolated data based on an 
output from one of the second and third inter- 
polation steps: and 

the output step of outputting an interpolated 
signal on the basis of interpolated data, which 
is based on an output from the first interpolation 
step, and the output interpolated data from the 
selection step. 



23. The medium according to claim 22, wherein on the 
basis of the determination result of the correlations 
in the first and second directions with respect to the 
signal at the pixel position as an object of interpo- 

5 lation, the output step comprises outputting an in- 
terpolated signal by weighting the interpolated data 
based on the output from the first interpolation step 
and the output interpolated data from the selection 
step. 

10 

24. The medium according to claim 23, wherein if it is 
expected to be determined that the correlations in 
the first and second directions would become equal, 
the output step comprises generating an interpolat- 
es ed signal by performing weighting such that the in- 
terpolated data based on the output from the first 
interpolation step is weighted more than the output 
interpolated data from the selection step. 

20 25. The medium according to claim 22, wherein the sec- 
ond interpolation step comprises performing inter- 
polation on the basis of pixel data, in the first direc- 
tion, over a broader range than that when the first 
interpolation step performs interpolation. 

25 

26. The medium according to claim 22, wherein the 
third interpolation step comprises performing inter- 
polation on the basis of pixel data, in the second 
direction, over a broader range than that when the 

30 first interpolation step performs interpolation. 

27. The medium according to claim 22, wherein said op- 
eration process program further comprises the em- 
phasizing step of emphasizing the output interpo- 

35 lated signal from the output step in accordance with 
the determination result of the correlations in the 
first and second directions with respect to the signal 
at the pixel position as an object of interpolation. 

40 28. The medium according to claim 27, wherein the em- 
phasizing step comprises emphasizing one of the 
first and second directions. 

29. The medium according to claim 22 , wherein the sec- 
45 ond and third interpolation steps comprise perform- 
ing interpolation by using HPFs (High Pass Filters). 

30. An operation process program for interpolating an 
output color signal from an image sensing device, 

so comprising: 

the first interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in first and 
55 second directions; 

the second interpolation step of performing in- 
terpolation in a pixel position as an object of in- 
terpolation on the basis of pixel data in the first 
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direction; 

the third interpolation step of performing inter- 
polation in a pixel position as an object of inter- 
polation on the basis of pixel data in the second 
direction; 5 
the selection step of determining correlations in 
the first and second directions with respect to 
a signal at a pixel position as an object of inter- 
polation and, on the basis of the determination 
result, selecting interpolated data based on an io 
output from one of the second and third inter- 
polation steps; and 

the output step of outputting an interpolated 
signal on the basis of interpolated data, which 
is based on an output from the first interpolation 15 
step, and the output interpolated data from the 
selection step. 



phasizing step comprises emphasizing one of the 
first and second directions. 

37. The program according to claim 30, wherein the 
second and third interpolation steps comprise per- 
forming interpolation by using HPFs (High Pass Fil- 
ters). 



31 . The program according to claim 30, wherein on the 
basis of the determination result of the correlations 20 
in the first and second directions with respect to the 
signal at the pixel position as an object of interpo- 
lation, the output step comprises outputting an in- 
terpolated signal by weighting the interpolated data 
based on the output from the first interpolation step 25 
and the output interpolated data from the selection 
step. 



32. The program according to claim 31 , wherein if it is 
expected to be determined that the correlations in 30 
the first and second directions would become equal, 
the output step comprises generating an interpolat- 
ed signal by performing weighting such that the in- 
terpolated data based on the output from the first 
interpolation step is weighted more than the output 35 
interpolated data from the selection step. 



33. The program according to claim 30, wherein the 
second interpolation step comprises performing in- 
terpolation on the basis of pixel data, in the first di- 40 
rection, over a broader range than that when the 
first interpolation step performs interpolation. 



34. The program according to claim 30, wherein the 
third interpolation step comprises performing inter- 45 
polation on the basis of pixel data, in the second 
direction , over a broader range than that when the 
first interpolation step performs interpolation. 



35. Theprogramaccordingtoclaim30,furthercompris- so 
ing the emphasizing step of emphasizing the output 
interpolated signal from the output step in accord- 
ance with the determination result of the correla- 
tions in the first and second directions with respect 
to the signal at the pixel position as an object of in- 55 
terpolation. 



36. The program according to claim 35, wherein the em- 
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